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(54) Method for enhancement of reduced color set images 



(57) A method of improving the range of color imag- 
es which can be processed by multiple enhancement 
processing elements, and improving image quality of 
limited color set images. The inventive method deter- 
mines (107) the number of colors that form an image, 



and with that information, adds a control function to one 
or more enhancement processors (112,122,132), 
which, based on a number of colors, increases and/or 
decreases the effect of the enhancement function. In do- 
ing so, the range of color images which can be handled 
is increased. 
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Description 



The present invention is directed toward a method 
and apparatus for improving the appearance of a digital 
image having a pictorial scene, and more particularly, 
toward improving images that are low color quality rep- 
resentations of natural scene images. 

In the past, a typical application for copiers or scan- 
to-print image processing systems was to reproduce an 
input image as accurately as possible, i.e. : render a 
copy. Thus, copies have been rendered as accurately 
as possible, flaws and all. However, as customers be- 
come more knowledgeable in their document reproduc- 
tion requirements, they recognize that an exact copy is 
often not what they want. Instead, they would rather ob- 
tain the best possible document output. Until recently, 
image quality from the output of a copier or a scan-to- 
print system was directly related to the input document 
quality. One very common set of input documents in- 
cludes photographs. Unfortunately, photography is an 
inexact science, particularly among amateurs, and orig- 
inal photographs are often poor. Alternately, technology, 
age or image degradation variations often result in pic- 
tures having an unsatisfactory and undesirable appear- 
ance. What is desired then, is a copy giving the best 
possible picture, and not necessarily a copy of the orig- 
inal. 

Image enhancement, as used herein, refers only to 
processes which improve the output quality of the im- 
age, and not to internal operations, necessitated by par- 
ticular processor limitations. Thus, for example, chang- 
ing TRC for better reproduction of an image is an en- 
hancement operation. However, reducing the number of 
colors representing image to place an image into a par- 
ticular file format, such as .GIF files, or .BMP files, is not 
an enhancement operation. 

Photography has long dealt with this issue. Analog 
filters and illumination variations can improve the ap- 
pearance of pictures in the analog photographic proc- 
ess. Thus, for example, yellow filters enhance the ap- 
pearance of white clouds against a blue sky in black and 
white images. Further, various electrophotographic de- 
vices, including digital copiers, can clean up and im- 
prove images by adjustment of threshold, filtering, or 
background suppression. Generally, these methods are 
manual methods which a user must select on an image 
by image basis. Unfortunately, the casual user is not 
skilled enough to perform these operations. The inability 
to perform image enhancement operations is exacer- 
bated when additionally dealing with color controls. 

Three possible choices are presented by the art in 
the area of image enhancement. In the first case, we 
can do nothing. Such a system is a stable system, in 
that it does no harm to an image. This is a common ap- 
proach taken to reproduction. However, the output doc- 
uments of such a system are sometimes not satisfactory 
to the ultimate customer. 

In a second case of image enhancement, the image 



can always be processed. It turns out that an improve- 
ment can usually be made to an image if certain as- 
sumptions are made that are accurate for most cases. 
In an exceptionally large set of images, increasing con- 
5 trast. sharpness, and/or color saturation, will improve 
the image. This model tends to produce better images, 
but the process is unstable, in that for multi-generation 
copying, increases in contrast, saturation, or sharpness 
are undesirable and ultimately lead to a severe image 
io degradation. Further, the process may undesirably op- 
erate on those images which are good ones. 

Accordingly, we arrive at our third case of image en- 
hancement, a process of automated image enhance- 
ment which operates to vary images which are not per- 
is ceived as good images, but does not operate on images 
which do not need to be improved. 

Many improvements can be made to an image, in- 
cluding luminance enhancement (e.g. US- A 5,450,502); 
sharpness enhancement (e.g., US-A 5,363,209); expo- 
20 sure adjustment (e.g. US-A 5,414 : 533); color balance 
correction (e.g., US-A 5.357.352. US-A 5,371.615) or 
contrast enhancement (US-A 5 : 581 : 370); color satura- 
tion correction (e.g. US-A 5,450,217). etc. These proc- 
esses can be used together in a predictive mode that 
25 does not require iterative processing (e.g. US A 
5,347,374). Generally, these processing methods oper- 
ate by modifying a set of reproduction curves (TRCs). 
The output image is achieved by using TRC curves, op- 
erating either on the luminance channel of an image ex- 
30 pressed LC^ C 2 coordinates, or preferably on each chan- 
nel in a color density space description of the image in 
Red-Green-Blue (rgb) coordinates. 

Unfortunately, the set of possible color images that 
can be enhanced is large, and varied. Problems are en- 
35 countered in using complex systems having multiple en- 
hancement processors as described, particularly where 
a prior process has reduced continuous tone images to 
an images having only limited number of colors. This 
limited color set images are often encountered in situa- 
40 tions where images are retrieved from image archives 
and in situations, where the images are used as back- 
grounds or pictograms. Images with a limited set of 
colors cause problems, because the process algorithms 
assume normal color distributions, but when confronted 
45 with limited color set. make inappropriate changes. 
These limited color set images, which are not strictly 
speaking natural color images, do not respond well to 
some of the enhancement operators. On the other hand, 
processor optimized for limited color set images do not 
so operate satisfactorily for natural color continuous tone 
images. Since low color set images provide only unreli- 
able image statistics, image processing operations have 
to be adapted to the statistical uncertainties and not full 
advantage can be taken of the processes when com- 
55 pared to the use of reliable image statistics. It would be 
highly desirable to determine a way to reliably use the 
same processors throughout a more substantial portion 
of the set of color images 
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In accordance with the present invention, a method 
of controlling an image enhancement processing com- 
prises creating a signal indicative of the number of 
colors in an image: and using said signal to modify the 
enhancement processing thereof. 5 

There is provided a method of improving the range 
of color images which can be processed by multiple en- 
hancement processing elements, and improving image 
quality of limited color set images. 

The inventive method determines the number of io 
colors that form an image, and with that information, 
adds a control function to one or more of the enhance- 
ment processors, which, based on a number of colors, 
increases and/or decreases the effect of the enhance- 
ment function. In doing so, the range of color images *5 
which can be handled is increased. 

These and other aspects of the invention will be- 
come apparent from the following descriptions used to 
illustrate the preferred embodiment of the invention, 
read in conjunction with the accompanying drawings in 20 
which: 

Figure 1 shows a block diagram of a system em- 
ploying the present invention; 

Figure 2 shows a block diagram of the processing 2s 
system described; and 

Figures 3 and 4 illustrate exposure control using the 
inventive concept. 

Referring now to the drawings where the showings 30 
are for the purpose of describing the embodiment of the 
invention and not for limiting same, reference is made 
to Figure 1 and the scan-to-print and archive-to-print 
system shown therein in which the present invention 
may find advantageous use. 35 

Figure 1 illustrates a scanner 1 0 which may conven- 
iently be a black and white or color scanner which pro- 
duces image signals defined in either RGB space for 
color images or density space for black and white imag- 
es. Alternatively images may come from an image ar- 40 
chive 11, where images are not necessarily idealized, 
and may represent unusual scenes or images. These 
images of concern are pictorial in nature, i.e.. they rep- 
resent "natural scenes", but they may be limited in their 
composition or may be problematic with their use. In- 
eluded in this discussion are images used as "back- 
ground " or "wallpaper" in computer displays. Another 
source of images are clipart archives, which represent 
large sets of inexpensively obtained images, which may 
not be readily available in an output that the user will so 
like. While certain computer generated imagery may 
qualify as representing nature scenes, the contemplat- 
ed images are predominantly scanned photographs. 
The images themselves are defined in terms of pixels, 
wherein each pixel has a gray value which varies be- 55 
tween a white level and a black level. In a currently de- 
sirable system, in which calculations may be done on 8 
bits of information, 256 levels of gray will be available 



for use When each separation in r.g.b space is used, 
this allows 24 bits color, and over 16 million colors. Pix- 
els are also identified in terms of position, i.e.. a pixel 
defines a unique area within the image, identified by its 
position in a scan line, and the scan line position in a 
page. Color is therefore represented by triplets of gray 
pixels for each color pixel in the image, each triplet of 
gray pixel defining the color in each separation, which 
together form the color pixel. 

Color may also be defined by an image file perhaps 
received from another source, using less than 256 levels 
of gray, and in some situations, allowing only 256 colors. 
Additionally, the number of describable colors/gray lev- 
els in an image is only the maximum number of possible 
colors and in a large number of cases, only a small sub- 
set of the allowed colors is actually used. This is often 
the case in images that are intended as background im- 
ages, pictograms, etc. or in images that were intended* 
for use on the World Wide Web. 

The described color images may be directed to an 
automated image enhancement system which will be 
further defined herein. For our purposes, the automated 
image enhancement system may include a segmenta- 
tion system which can identify within a document a type 
of image or image object, including pictorial and non- 
pictorial image areas or objects. It will be assumed that 
the output of the automated image enhancement sys- 
tem that is described herein will be directed to a printer, 
CRT. or like device. These devices may have many dis- 
tinct characteristics and may be laser printers, or ink jet 
printers or LED displays or CRT displays. However, they 
have as a common requirement the representation of 
gray pictorial images. This may be done with gray print- 
ing or pseudo gray printing. 

In terms of deriving data for operation by the present 
image enhancement system, a prescan may be per- 
formed on a document placed on a copying platen and 
scanned by the electro-optical system of the scanner to 
produce a signal representing the document image. Al- 
ternatively, the image may be directed to the automated 
image enhancement system from a memory, having 
previously been scanned or derived some other system, 
in which case, the received image is sampled as re- 
quired. The prescan is optionally undersampled, i.e., the 
image need not be sampled at the ultimate resolution of 
the system for the purposes of enhancement. In prac- 
tice, it has been determined that a relatively small 
number of pixels representative of and dispersed 
through the entire image can accurately represent the 
image for this purpose. In one particular embodiment, 
we use a block of pixels derived from the image, in ap- 
proximately 512 pixels x 512 pixels. The primary pur- 
pose of this selection is to improve the speed at which 
a software image enhancement system can process the 
pictorial images. Sampling at common image resolu- 
tions does not improve the results noted in the inventive 
process herein described significantly, and dramatically 
increases the software processing time required. Some 
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embodiments of the described inventive process might 
decide not to undersampte the image, eg , when the 
input image size is actually small, or when a hardware 
acceleration is used. 

The input image color signals are edited, or en- 
hanced and prepared for printing at color image 
processing unit 20, which in Figure 1 provides as an out- 
put r, g, b signals. Printer 30 receives the signal from 
color image processing unit 20. and provides for con- 
version of the signals to printer driver signals, commonly 
given in terms of the colorants that will be used in repro- 
duction, typically Cyan, Magenta, Yellow and Key or 
black (CMYK). The printer has an output of the natural 
scene image, which, hopefully, matches user expecta- 
tions of a "good" output image. Alternatively, the proc- 
essed image from color image processing unit 20 is dis- 
played on a television screen, CRT display or the like. 

For the description of the remainder of the process 
of the invention, reference is made to the image of Fig- 
ure 2. Data channel 100 is a bus which carries image 
data, conveniently in terms of rgb color density signals. 
Enhancement channel 102 carries data from data chan- 
nel 100 to subsampler 104, which as described, option- 
ally undersamples the data and delivers it to information 
generation and distribution 1 06 which accomplishes the 
common functions required for all the enhancement 
modules, and which may conveniently be derived at the 
same time. Accordingly, information generation and dis- 
tribution 106 accomplishes conversion of the image 
from RGB space to luminance space - or an approxima- 
tion thereof, derives histograms of signals as required, 
globally and/or locally, and determines such other data 
as required by any enhancement operators used in the 
system. 

The color image data initially received from scanner 
10 or the like, is assumed to be in RGB space initially, 
i.e., red-green-blue space, and for the inventive proc- 
ess, must initially be converted to luminance space (e. 
g. Y in YC^ space : a similar luminance/chrominance- 
like value, or an approximation thereof) at information 
generatbn and distribution 106. It is possible that the 
image will already be in luminance space, as it is com- 
mon to convert RGB values to luminance/chrominance 
space for other image processing. YC A C 2 space is a 
useful space in which the inventive process can be per- 
formed, and Xerox YES space is one possible embodi- 
ment of such a space. Whatever space is used must 
have a component which relates to the human visual 
perception of lightness or darkness, such as Y of Xerox 
YES of the "Xerox Color Encoding Standard,- XNSS 
289005, 1 989. In the following, the invention will be de- 
scribed using the Xerox YES color space, although it is 
not limited thereto. 

In addition to the above described operation, at 
block 107, the colors present in the image are counted. 
Generally speaking, continuous tone images will have 
a large number of colors, and limited color set images 
will have less than e.g.: 256 colors. We have selected 



256 as a reasonable threshold for the operations that 
will be described. It will be appreciated that it is possible 
to represent limited cobr set images in 24 bit color 
space, but the image will not use all of the possible 
5 colors. On reading the data, a count of the different 
colors is kept, optionally including a color/Black & White 
indicator. A count value, or an indication of whether a 
count exceeds a preselected threshold is output by color 
counter 107. 

w From information generation and distributbn 106, 
data is made available on information channel 103, 
which carries multiple signals to each correction module 
buses 1 10 : 1 20 and 1 30, including the information about 
the number of colors in the image and the optional color/ 
'5 Black & White indicator. Correction processor buses 
110, 120 and 130. respectively, forward signals required 
for operation, for example, only, to exposure processor 
112, color balance processor 122 and contrast proces- 
sor 1 32, and likewise return information to information 
channel 1 08 as required. Each of exposure processor 
112. color balance processor 122 and contrast proces- 
sor 1 32 use the data supplied via information channel 
1 08 to calculate the signals for TRC generators 1 1 4, 1 24 
and 134 respectively, to generate TRC's reflecting re- 
quired image modification determined at each proces- 
sor. However, in accordance with the invention, the out- 
put of each of the processors is altered based on the 
number of colors found in the image being processed 
as received from 1 06. Each of TRC generators 11 4. 1 24 
and 1 34 directs generated TRC's to TRC combiner 1 40. 
which combines the generated TRC's into a single look 
up table and stores the results at TRC controller 1 50 via 
bus 160. TRC controller 150 applies the changes to the 
rgb signals on bus 1 00 as required. Judge or arbiter 1 70 
operates to resolve conflicts in the TRC determination, 
so that processors 112, 122 and 132 do not operate at 
cross purposes. The output of TRC controller 1 50 might 
be directed to a sharpness adjustment stage, such as 
that described in US-A 5,363,209. The output of the 
TRC controller 1 50 might have a number of bits different 
from the original number of bits in the image. This is spe- 
cially useful if the output is directed to a sharpness ad- 
justment stage that optionally incorporates a reduction 
in the number of bits, say from 10 bits to 8 bits. 

It will, of course, be appreciated that the principle of 
the invention applies to any enhancement operation, 
and thus, while only three enhancement processors are 
illustrated here, additional processors or different com- 
binations of process, as described in the background, 
may be used. 

Exposure processor 112 and contrast processor 
132 both operate on the luminance histogram, and pref- 
erably operate in order of exposure, first, and contrast, 
second. Accordingly, after directing the exposure signal 
to TRC generator 114, the exposure signal produced at 
exposure processor 110 is directed to information gen- 
eration and distribution 1 06 via information channel 1 03. 
This signal is used to generate a modified histogram ac- 
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cording to the determined exposure adjustment. This 
modified histogram is subsequently filtered with a blur 
or smoothing function, such as an averaging over about 
10 adjacent levels, to blur or smooth peaks naturally re- 
sulting in the modified luminance histogram due to ex- s 
posure adjustment. It should be noted that the operation 
of this filter can be modified for increased or decreased 
smoothing. It also appears to be desirable to modify the 
filter operation based on the amount of correction to the 
histogram at the exposure processor. The blurred mod- io 
ified histogram is then forwarded to contrast processor 
132. via information channel 108 and correction proc- 
essor bus 1 30. 

In one possible implementation of the invention, a 
map of sensitivity of the respective analyzers/proces- '5 
sors, (an evaluation that represents the likelihood of im- 
proper processing if the image file is a reduced color set 
image for the used analyzer/processor combinations) 
has determined the following order, from most sensitive 
to least sensitive: 20 

1. color balance 

2. sharpness 

3. saturation 

4. contrast 25 

5. exposure 

For a color implementation, the processor outputs 
are modified with an additional "color map processing 
table" (available in each processor and responsive to 30 
the color count value) that governs the processing of 
mapped images. Alternatively, the processor might be 
maintained, but modified data, resulting in the desired 
operation, is outputted from the analyzer. In this table, 
the following processing is encoded: 35 

C1. Color Balance: 

For images of 256 colors and less, the color balance 
processing unit is deactivated. 40 

02. Sharpness: 

For images of 256 colors and less, the sharpness 
processing is limited to 25% ol normal operation, and at 45 
128 colors all sharpness processing is deactivated. 

C3. Saturation: 

For images of 256 colors or less, saturation so 
processing is limited to 50% of normal operation and at 
128 colors all saturation processing is deactivated. 

C4. Contrast: 

ss 

For images of 256 colors or less, contrast is 
switched to "save mode processing" At 128 colors 
processing is deactivated. 



C5. Exposure: 

For images of 64 colors and less, the processing is 
limited to 75% of normal operation, and at 16 colors 
processing is deactivated. 

For black & white images, the corresponding 
scheme is used. Here only contrast, exposure and 
sharpness are modified and an example for the settings 
is: 

G1. Exposure: 

For images with less than 64 levels of gray, expo- 
sure enhancement is reduced to 75% 

G2. Sharpness; 

For images with less than 64 levels of gray, sharp- 
ness enhancement is less than 25%, and at less than 
32 levels of gray, sharpness is turned off. 

G3. Contrast: 

For images of 64 levels of gray or less, contrast is 
switched to "save mode processing" (see, US A 
5,450,502: processing of images with high global vari- 
ances, specifically incorporated herein by reference), 
and at less than 32 levels of grays, contrast enhance- 
ment is turned off. 

In looking at one example of how this process limi- 
tation is obtained, reference is made to US 5.363,209 
to Eschbach and the changes which would be imple- 
mented therein. Figures 3 and 4 show a flow chart of the 
inventive process at step 300, r, g, b data is received. 
At step 302, the r, g, b data is converted to YES values. 
At step 304, histogram signal H(l) is derived from the Y 
signal. At steps 306 and 308 the process branches into 
processing the 100% range and the 90% range of H(l). 
Between steps 308, 310 and 31 2 the maximum value of 
H(l), I lighUOO, and the minimum value of H(l). I dark100, 
is compared respectively to threshold Tlight and T dark 
to determine whether to continue processing the image. 
Depending on the response, (noting the "yes" response 
of step 312 and the "no" response of step 310 that end 
the process) a value for gamma is derived by evaluating 
equation 12 at steps 322 for the case of the image being 
too light and 320 for the case of the image being too 
dark. At step 324 the preliminary values for gamma 100 
and gamma 90 are compared to find a disagreement, 
where a disagreement is reached when the difference 
between gamma 90 and gamma 100 is larger than a 
preset value DELTA gamma, where a DELTA gamma 
value of 0.5 has been found to give good results in the 
embodiment outlined in Figures 3 and 4. If a disagree- 
ment is detected, the gamma 90 value is ignored and 
only the gamma 100 value is used for further process- 
ing. At step 326, the gamma 100 value is compared to 
a preset value In this case gamma 100 <i~ 2 has been 
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found to give good results. However, in accordance with 
the invention, variations in gamma 100 are limited to 
gamma 1 00 </= 1 . 25 lor reduced color set images (colors 
where the count is less than 64, in the example) where 
gamma 100 is 1 provides a no-op. or no operation. This 5 
value is set in the process, based on the counted color 
value. The remaining process proceeds as in US-A 
5.41 4.538. It should be noted that the system described 
in Figure 2 is one possible implementation of a cascad- 
ed image enhancement system making use of the de- '0 
scribed inventive method. An alternate approach could 
locate the inventive method of altering the response of 
the enhancement processors in individual modules, 
which operate independently upon the image. 

It will no doubt be appreciated that the present in- '5 
vention may be accomplished with either software, 
hardware or combination software hardware implemen- 
tations. 

20 

Claims 

1. A method of controlling an image enhancement 
processing, comprising the steps of: 

25 

creating a signal indicative of the number of 
colors in an image; and 
using said signal to modify the enhancement 
processing thereof. 

30 

2. The method as defined in claim 1 , and including the 
step of: 

determining whether the image is a black and 
white image, or a color image; and, 35 
controlling said enhancement processing mod- 
ification in accordance with said determination. 

3. The method as defined in claim 1 or claim 2, where- 
in a plurality of distinct enhancement processing op- 40 
tions are available, and including: 

differentially controlling said distinct enhance- 
ment processing options, using said signal. 

45 
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(54) Method for enhancement of reduced color set images 



(57) A method of improving the range of color imag- 
es which can be processed by multiple enhancement 
processing elements, and improving image quality of 
limited color set images. The inventive method deter- 
mines (107) the number of colors that form an image, 



and with that information, adds a control function to one 
or more enhancement processors (112,122,132), 
which, based on a number of colors, increases and/or 
decreases the effect of the enhancement function. In do- 
ing so, the range of color images which can be handled 
is increased. 



NATURAL 
SCENE 
IMAGE 



10 



SCANNER 



r.g.b 



20 



COLOR 
IMAGE 
PROCESSING 



30 



PRINTER 
(CMYK) 



NATURAL 
SCENE 
REPRO- 
DUCTION 



3 FIG. 1 

N 

00 

CO 
O 
Q. 

LU 

Prnted by Jouve. 7500 1 PARIS (FR) 



BNSDOCID: <EP 0878777A3J_> 



EP 0 878 777 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Mumper 

EP 98 30 3591 



DOCUMENTS CONSIDERED TO BE RELEVANT 



aiegcry. 



Citation oi .-loajniern „viin matcatior. A.iere apofcpnate 
ci fgievant passages 



! Relevant 
| to claim 



CLASSIFICATION OF THE 
APPLICATION (lnt.CI.6) 



!EP 0 338 877 A (KONISHIROKU PHOTO IND) 
[ 26 September 1990 
I ▼ abstract * 

|* page 2. line 40 - page 3, line 16 * 
I * page 4. 1 ine 14 - line 37 * 

EP 0 413 557 A (SONY CORF) 
20 February 1991 

* abstract * 

:* column 2. line 5 - line 27 * 

I* page 5, line 26 - line 42: figure 4 * 

•PATENT ABSTRACTS OF' JAPAN 
vol. 095. no. 003. 29 September 1995 
!& JP 07 131807 A (NIPPON H0S0 KY0KAI 
! :0THERS: 01 5 : L9 May 1995 

* abstract * 



I he presert search report has been crawn up for 3li claims 



1-3 



1-3 



1-3 



G06T5/40 



| G06T 



TECHNICAL FIELDS 
SEARCHED (lnt.Cl.6i 



Ptace ot i«aicn 

THE HAGUE 



L-ue c: completion n we wiicn 

29 October 1998 



Gonzalez Ordonez, 0 



CATEGORY OF CITED DOCUMENTS 

oamcuiartv r*i»vantrf ta*«n atone 
Dan-c-jiafiy r#wart * ccmomea v/nn anciner 
Gucuin*tUol iih same category 
l^cnnologicai Cackjraurd 
■ pcn-wnaen c;scto:.ur» 
intarmeoiaw document 



T ■ irv»cry or pnncpls underlying in* inv«r*ior. 
E ■ #an<er patent rocum&n ou 3jcusr\*a 3". & 

a*T«r :no filing aate 
C document ;i:«o in tne application 
L decuman ated lor other reasons 

A m4frt;4r cf the iane p.venr tamilv cr-'respsncmg 

dOClirrvani 



2 



